How Accurately can Suborbital Experiments Measure the CMB? 
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Great efforts are currently being channeled into ground- and balloon-based CMB experiments, 
mainly to explore anisotropy on small angular scales and polarization. To optimize instrumental 
design and assess experimental prospects, it is important to understand in detail the atmosphere- 
related systematic errors that limit the science achievable with new suborbital instruments. For this 
purpose, we spatially compare the 648 square degree ground- and balloon-based QMASK map with 
the atmosphere-free WMAP map, finding beautiful agreement on all angular scales where both are 
sensitive. This is a reassuring quantitative assessment of the power of the state-of-the-art FFT- and 
matrix-based mapmaking techniques that have been used for QMASK and virtually all subsequent 
experiments. 
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I. INTRODUCTION 



Since the discovery of Cosmic Microwave Background 
(CMB) temperature anisotropy by COBE/DMR [1], a 
variety of ground-based and balloon-borne (or suborbital) 
experiments have detected and characterized the fluctu- 
ations in the CMB on different angular scales. These 
suborbital observing locations were chosen for being less 
costly than space-based alternatives, but at a price: none 
of these experiments could enjoy the level of systematic 
control and calibration accuracy that has been achieved 
in space missions such as COBE and WMAP [2]. To deal 
with the contamination from both atmospheric emission 
and various other offsets and modulations related to the 
warmer and less stable suborbital environment, a power- 
ful set of data analysis tools were developed. These come 
in many names and guises (see, e.g., [3-18]), but all do 
roughly the same thing. First the raw time-ordered data 
is cleaned both in real space (removing segments of low- 
quality data) and in Fourier space (using notch filters to 
remove, e.g., scan-synchronous offsets and balloon pen- 
dulation lines) and approximately prewhitened with FFT 
techniques to better treat correlated noise and detector 
response. Then the time-ordered data is compressed into 
a pixelized sky map by inverting a large matrix, either by 
brute force, iteratively or using some approximation, and 
the corresponding noise covariance matrix is calculated 
or estimated. Numerous tricks for numerically acceler- 
ating this second step are available using fast Fourier 
transforms, circulant matrices or Toeplitz matrices. 

Although they are mathematically elegant, there have 
been few precision tests of how well these methods work 
in practice for suborbital experiments. The numerical 
methods on which they are based only guarantee that 
they can accurately deal with precisely those problems 
that they were designed to deal with, so it is crucial 
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FIG. 1. Visual comparison of the QMASK (left) and WMAP 
(right) maps, both Wiener filtered with the same weighting (deter- 
mined by the QMASK noise covariance matrix). The slightly larger 
fluctuation level seen in the left image gives a visual indication of 
the effect of QMASK noise. 

to perform end-to-end tests that are able to detect un- 
foreseen problems as well. By far the most powerful test 
(in the statistical sense of being more likely to discover 
systematic errors) involves a direct comparison of the sky 
maps from experiments that overlap in both spatial and 
angular coverage, rather than merely a comparison of 
the measured angular power spectra. Some of the best 
testimony to the quality of CMB maps comes from the 
success of such map comparisons in the past — between 
FIRS and DMR [19], Tenerife and DMR [20], FIRAS and 
DMR [21], MSAM and SK (Saskatoon) [22], QMAP and 
SK [23], QMASK and DMR [24], PIQUE and SK [25], 
POLAR and DMR [26], DMR and WMAP [2], Maxima 



and WMAP [27], BOOMERanG and WMAP [28], Tener- 
ife and WMAP [29], two years of Python data [30], four 
years of DMR data [31], three years of SK data [32], two 
flights of MS AM [33], and different channels of QMAP 
[7-9], BOOMERanG [11], MAXIMA [12], and WMAP 
[18]. It has now become possible to perform such tests 
with unprecedented sensitivity, using the spectacularly 
clean and well-tested WMAP data [2] as the answer sheet 
against which to compare other maps. So far, this has 
only been done carefully (beyond simple visual compar- 
isons) for MAXIMA, with encouraging results [27]. This 
work showed that the WMAP-MAXIMA cross power 
spectrum was consistent with the power spectra mea- 
sured from the two experiments separately. 

The purpose of this letter is to carry out such a test for 
the QMASK data, which was the largest publicly avail- 
able degree-scale CMB map prior to the WMAP release, 
and covers more than six times the area of the MAXIMA- 
I map. Not only was this map created using the above- 
mentioned Fourier and matrix techniques, but it com- 
bines measurements from both the ground-based SK ex- 
periment [5] and two balloon-based data sets from the 
QMAP experiment [7-9] , therefore being potentially vul- 
nerable to almost any systematic error one could imagine. 

Although the two maps have different angular reso- 
lutions, their comparison is straightforward since the 
WMAP data is effectively noiseless (compared to the 
QMASK map) and can easily be smoothed to the 
QMASK angular resolution. Below we compare these 
two data sets using four different statistical tests, then 
present our conclusions in Section III. 



II. METHODS & RESULTS 

A. Data sets used in the analysis 

The QMASK map was constructed from the combina- 
tion of the SK and QMAP data sets [24] . It consists of 
6590 pixels arranged on a simple square grid in gnomonic 
equal area projection, and its angular resolution corre- 
sponds to a Gaussian beam with FWHM 0.68°. These 
pixels are conveniently arranged in a 6590-dimensional 
vector X whose noise covariance matrix Nqmask = (xx*) 
is known and available online [24]. We compare it with 
both the WMAP Q-band map [2] and a foreground- 
cleaned CMB map made from a combination of the five 
WMAP CMB maps [34], hereafter TOH03. Before per- 
forming any analysis, the WMAP-Q and TOII03 maps 
were smoothed to have the same net beam function as 
QMASK, and data that do not overlap the QMASK ob- 
serving region were discarded. 



B. Visual Comparison 

We begin with a simple visual comparison. Comparing 
plots of the raw maps of QMASK (x) and WMAP (y) 
is not particularly useful, since QMASK is a noisy data 
set. Instead, we compare Wiener-filtered versions of both 
maps: 
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where 
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and S is the contribution to the map covariance ma- 
trix from CMB signal. The WMAP noise is so much 
smaller than that of QMASK that its exact value is irrel- 
evant for our applications, so we make the approximation 
NwMAP ~ 0, i.e., 



W«S[S + N 



QMASKj 



(3) 



This linear filtering maximizes the signal-to-noise ra- 
tio in the QMASK map by effectively smoothing the 
map and downwcighting noisy pixels and modes. Equa- 
tion (1) shows that to ensure a fair comparison, we 
Wiener-filter the WMAP map using the exact same W- 
matrix as for the QMASK map. The Wiener filtered 
QMASK map x'" (left) and WMAP TOH03 map y'" 
(right) arc shown in Figure 1, and are seen to look encour- 
agingly similar. Even the "anomalous" cold-spot around 
(RA,DEC)=(3''20™, 84°55') discussed in the QMASK 
analysis [24] (also seen by SK [35] but not by DMR [24]) 
is clearly visible in the WMAP data. The results are 
found to depend only weakly on the detailed choice of S. 
We wish to keep our map comparisons as theory-free as 
possible, and therefore choose S corresponding to a sim- 
ple scale-invariant spectrum with 6Tf = SOiiK here and 
below. 



C. Cross-correlation results 

Next, we compare the QMASK and WMAP maps by 
cross-correlating them. The measured cross-correlation 
coefficient a and its variance Aa are given by [36] 



a = 



y^c-iy' 



Aa2^(a2)-(a)2=[yTc-iy]-\ 
where the covariance matrix C is defined as 

C = N + S = Nqmask + S. 



(4) 
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As a first reality check, we cross-correlate the full 
QMASK data with the TOII03 map, obtaining a = 



0.921 ±0.061. This provides an encouraging validation of 
the QMASK observations, showing a detection of CMB 
signal at about the 15a level. This also confirms and fur- 
ther strengthens the results of [24], who cross-correlated 
QMASK and DMR data [1,31] and concluded that the 
CMB signal was consistent between the two data sets. 
It also provides a completely independent test of the ac- 
curacy with which the QMAP and SK experiments were 
calibrated, showing that they got things right to well 
within the quoted 10% cahbration uncertainty [24]. 

Using only the sky area above |&| > 20° (a Galac- 
tic cut), we detect the CMB signal at the 14.7a level 
(a = 0.938 ± 0.064), and for a |6| > 30° Galactic cut 
the detection is around the 9a level (a — 0.950 ± 0.110). 
The similarity between the 20° and 30° cut results also 
confirms the conclusion of [37] that QMASK foregrounds 
are not dominant at high Galactic latitudes (5 >20°). 



D. The x^-test 




Relative Normalization r 



A simple and robust way to test for discrepancies be- 
tween the QMASK and WMAP maps is to calculate x^ 
defined by 



X 



.^N- 
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QMASK' 



(7) 



where z = ax — y is the difference of the two maps after 
adjusting the QMASK calibration to the best-fit value 
found in the previous section. In the absence of system- 
atic or calibration errors, the corresponding difference 
map should contain pure noise, so we should have (z)=0 



and X l^p 



1, where Nr, 



6590 is the number of 



pixels in the QMASK map vector x. This is precisely 
what we find: we obtain x^ /Npix ~ 1.00, i.e., the dif- 
ference map z is consistent with pure noise. This means 
that the strong signal seen in Figure 1 is common to both 
maps and due to a true sky signal, with no evidence for 
significant systematic errors in QMASK. 



E. The Null-buster test 

How statistically significant is our detection of signal 
in the maps? Consider the null hypothesis that a map 



(-0 



N. Given the 



z contains merely noise, i.e.. 

alternative hypothesis (zz-^) = N -I- S, one can show that 
the most powerful "null-buster" test for ruling out the 
null hypothesis is using the statistic [38,24] 



z^N-^SN-^z-trN-iS 



[2tr {N-iSN-iS}] 



1/2 



(8) 



which can be interpreted as the number of "sigmas" at 
which the null noise-only hypothesis is ruled out. When- 
ever we have reason to suspect systematic errors of a 
certain form are producing a signal oc S, the null-buster 
test is more sensitive to these systematic errors than the 



FIG. 2. NuU-bustcr comparison of QMASK (left) and 
WMAP-Q (right). This figure shows the number of standard de- 
viations ("sigmas") at which the difference map z = x — ry is 
inconsistent with pure noise. 



X^-test described in the previous subsection, which is a 
general-purpose tool. 

The presence of foreground contamination (see [39] for 
a review) has been quantified independently for both the 
SK [36] and QMAP [37] experiments, by cross-correlating 
them with various foreground templates. The contamina- 
tion level was found to be small enough to have negligible 
impact on the measured QMASK power spectrum [40], 
but as we will see below, it is nonetheless large enough 
to affect the more sensitive test we are performing here. 
The main QMASK foregrounds are synchrotron radia- 
tion, free-free emission and perhaps spinning dust emis- 
sion, all of which become more important (relative to 
the CMB) at lower galactic latitudes and on larger an- 
gular scales (low t). To focus on the CMB signal. Figure 
2 therefore shows the nuUbuster test results limited to 
\b\> 30° and angular scales € > 30. The latter is achieved 
by summing only over I = 30, ..., 200 when computing S 
in equation (8) (beam issues limit QMASK to t < 200 
[40] ) . This also corresponds to the angular range in which 
QMASK carries the bulk of its information, since limited 
sky coverage and angular resolution limit the information 
at ^ < 30 and i > 200, respectively. 

Figure 2 shows that whereas QMASK (left; r = 0) and 
WMAP-Q (right; r = oo) detect sky signal at the level 
of about 30(T and 2300cr, respectively, the difference map 
(corresponding to r=l) is consistent with pure noise, and 
the null hypothesis of no systematic errors fails at less 



than 2cr level.* The fact that the curve has such a narrow 
minimum at r ~ 1 also provides a strong confirmation 
that the relative caHbration of QMASK and WMAP is 
good. 

We used the WMAP-Q map for this test because it 
is closest in frequency to that of the QMASK map, 
which is constructed from a combination of Ka- and 
Q-band data. We also repeated the nuUbuster test us- 
ing the foreground-cleaned TOH03 WMAP map, finding 
marginally worse agreement as expected, since this com- 
pares a map that includes foregrounds with an essen- 
tially foreground- free map. As expected, the agreement 
between the two maps also deteriorated when including 
data at lower Galactic latitudes, where foregrounds be- 
come more important. 



III. CONCLUSIONS 

In summary, a powerful set of FFT- and matrix-based 
tools have been developed in recent years for producing 
quality CMB maps from time-ordered observations. Al- 
though these tools are mathematically elegant, there have 
been few precision tests of how well they work in practice 
when faced with real-world systematic errors from sub- 
orbital experiments. We have performed a precision test 
of precisely this, comparing the QMASK map with the 
"answer sheet" provided by WMAP. The results are very 
encouraging, showing excellent agreement and no statis- 
tically significant evidence of residual systematic errors. 
This suggests that although space-based observations will 
obviously be required for many tasks, e.g., all-sky map- 
ping of CMB polarization, suborbital experiments have 
the potential to continue playing a key role in the preci- 
sion cosmology era. 
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